Robots.txt Disallow command [on hold]

Posted by Saahil Sinha on Pro Webmasters See other posts from Pro Webmasters or by Saahil Sinha
Published on 2013-06-23T02:47:18Z Indexed on 2013/06/28 4:29 UTC
Read the original article Hit count: 343

Filed under:

search-engines

|

robots.txt

|

duplicate-content

How to disallow folders through Robots.txt, which are been crawled due to wrong url structure, which thus cause duplicate page error

The URL been crawled as incorrectly by Google leading to duplicate page error:

www.abc.com/forum/index.php?option=com_forum

However, The actual correct pages however are:

www.abc.com/index.php?option=com_forum

Is this a correct way by excluding them through robots.txt: To exclude

 www.abc.com/forum/index.php?option=com_forum

Below is command

Disallow: /forum/

Will it not block in legitimate component folder 'Forum' of site?

© Pro Webmasters or respective owner

Related posts about search-engines

Your Content Can Get Visibility Through Search Engines - Part 2

as seen on Ezine Articles - Search for 'Ezine Articles'
Your content plays an important role in defining the success of your search engine marketing plan and when you talk about SEO in India, and then you will easily understand different kinds of discoveries and how to use them for your benefit. After you have understood the traditional discovery and some… >>> More
Your Content Can Get Visibility Through Search Engines - Part 1

as seen on Ezine Articles - Search for 'Ezine Articles'
Search engine marketing aims at getting higher search engine rankings for your website but for that you need to get yourself into the search engine Index first. Your content designed for your SEO activity should be accessible to millions of internet users and if your content is not found through the… >>> More
Most Popular Internet Search Engines - Four Tips to Raise Your Website's Search Engine Ranking

as seen on Ezine Articles - Search for 'Ezine Articles'
When you've got a website, one of the most important factors you need to consider is search engine rankings. If you don't have a great ranking position, you won't be easy to find. Many people will only look through the first five pages of search engine results. For that reason, your website must be… >>> More
Tip/Trick: Fix Common SEO Problems Using the URL Rewrite Extension

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Search engine optimization (SEO) is important for any publically facing web-site. A large % of traffic to sites now comes directly from search engines, and improving your site’s search relevancy will lead to more users visiting your site from search engine queries. This can directly or… >>> More
How Does WordPress Blog Search Engines?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, If you go to wordpress admin-settings-privacy, there are two options asking you whether you want to allow your blog to be searched though by seach engines and this option: I would like to block search engines, but allow normal visitors How does wordpress actually block search bots/crawlers… >>> More

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More